Feature: Book Genie Mode - Automated Testing with Detailed Reporting
File name: BookGenie.feature
Relative path: features/BookGenie.feature
Scenario Outline: Book Genie Response Validation with Citation Verification
When
I open the Creative Workspace login page
00:00:02.988
Then
I should see the homepage
00:00:00.035
When
I click on the mode selection dropdown
00:00:00.000
Then
I can see the "BookGenieQA" mode
00:00:00.000
And
I select the "BookGenieQA" mode
00:00:00.000
Then
I type "Suggest 5 books on christmas" on chat input element
00:00:02.971
And
I wait for AI to complete thinking
00:00:43.479
And
I validate the response is visible for "Suggest 5 books on christmas"
00:00:00.010
And
I extract book data from BookGenie response
00:00:00.009
And
I save BookGenie book data to Excel file for query "Suggest 5 books on christmas"
00:00:00.011
And
I validate each book individually against Excel file for query "Suggest 5 books on christmas"
00:00:00.013
CITATION VALIDATION DETAILED REPORT ================================================================================ BOOK: Finding Christmas ------------------------------------------------------------ ✅ Reason 1: PASS (95%) ✅ Reason 2: PASS (100%) ✅ Reason 3: PASS (95%) BOOK: How the Dukes Stole Christmas ------------------------------------------------------------ ✅ Reason 1: PASS (95%) ✅ Reason 2: PASS (100%) ✅ Reason 3: PASS (100%) BOOK: The First Christmas ------------------------------------------------------------ ❌ Reason 1: FAIL (0%) 📝 Reason: The book is titled The First Christmas... 📚 Citation: Book Title: The First Christmas... ⚠️ Citation text not found in reason: "Book Title: The First Christmas" ✅ Reason 2: PASS (100%) ❌ Reason 3: FAIL (0%) 📝 Reason: The book treats the stories of the first Christmas... 📚 Citation: The idea to write this book about the stories of the first Christmas... ⚠️ Citation text not found in reason: "The idea to write this book about the stories of the first Christmas" BOOK: Christmas 101 ------------------------------------------------------------ ❌ Reason 1: FAIL (0%) 📝 Reason: The book is titled Christmas 101... 📚 Citation: Book Title: Christmas 101... ⚠️ Citation text not found in reason: "Book Title: Christmas 101" ✅ Reason 2: PASS (100%) ✅ Reason 3: PASS (100%) BOOK: So This Is Christmas ------------------------------------------------------------ ❌ Reason 1: FAIL (0%) 📝 Reason: The book is titled So This Is Christmas... 📚 Citation: Book Title: So This Is Christmas... ⚠️ Citation text not found in reason: "Book Title: So This Is Christmas" ✅ Reason 2: PASS (100%) ✅ Reason 3: PASS (100%) SUMMARY: 11/15 reasons passed (73.3%) OVERALL STATUS: FAIL
Citation validation failed: 11/15 passed (73.3%)
Citation Validation Summary: 11/15 passed
📊 Citation Validation Report
Generated on 10/17/2025, 6:17:36 PM
📖 Finding Christmas
✅ Reason 1: 95% match
✅ Reason 2: 100% match
✅ Reason 3: 95% match
📖 How the Dukes Stole Christmas
✅ Reason 1: 95% match
✅ Reason 2: 100% match
✅ Reason 3: 100% match
📖 The First Christmas
❌ Reason 1: 0% match
⚠️ Citation text not found in reason: "Book Title: The First Christmas"
✅ Reason 2: 100% match
❌ Reason 3: 0% match
⚠️ Citation text not found in reason: "The idea to write this book about the stories of the first Christmas"
📖 Christmas 101
❌ Reason 1: 0% match
⚠️ Citation text not found in reason: "Book Title: Christmas 101"
✅ Reason 2: 100% match
✅ Reason 3: 100% match
📖 So This Is Christmas
❌ Reason 1: 0% match
⚠️ Citation text not found in reason: "Book Title: So This Is Christmas"
✅ Reason 2: 100% match
✅ Reason 3: 100% match
📈 Validation Summary
Total Reasons Validated: 15
Passed: 11
Failed: 4
Pass Rate: 73.3%
Overall Status: FAIL
Then
I verify database connectivity
00:00:00.358
And
database should contain 10032 books
00:00:00.170
And
I validate extracted books individually against database
00:00:00.771
And
I validate BookGenie response relevance with AI for query "Suggest 5 books on christmas"
00:00:06.246
+ Show Info
+ Show Info
PER-BOOK AI VALIDATION REPORT - Query: "Suggest 5 books on christmas"
================================================================================
OVERALL RELEVANCE SCORE: 70%
STATUS: FAIL
BOOKS ANALYZED: 5
BOOK 1: "Finding Christmas"
------------------------------------------------------------
Overall Score: 70%
SECTION SCORES:
Author Information: 70% - Analysis unavailable - API error
Publishing Date: 70% - Analysis unavailable - API error
Why Match Explanations: 70% - Analysis unavailable - API error
Relevance Scores: 70% - Analysis unavailable - API error
DETAILED FEEDBACK:
• AI analysis temporarily unavailable due to service overload
IMPROVEMENT SUGGESTIONS:
• Retry analysis when service is available
================================================================================
BOOK 2: "How the Dukes Stole Christmas"
------------------------------------------------------------
Overall Score: 70%
SECTION SCORES:
Author Information: 70% - Analysis unavailable - API error
Publishing Date: 70% - Analysis unavailable - API error
Why Match Explanations: 70% - Analysis unavailable - API error
Relevance Scores: 70% - Analysis unavailable - API error
DETAILED FEEDBACK:
• AI analysis temporarily unavailable due to service overload
IMPROVEMENT SUGGESTIONS:
• Retry analysis when service is available
================================================================================
BOOK 3: "The First Christmas"
------------------------------------------------------------
Overall Score: 70%
SECTION SCORES:
Author Information: 70% - Analysis unavailable - API error
Publishing Date: 70% - Analysis unavailable - API error
Why Match Explanations: 70% - Analysis unavailable - API error
Relevance Scores: 70% - Analysis unavailable - API error
DETAILED FEEDBACK:
• AI analysis temporarily unavailable due to service overload
IMPROVEMENT SUGGESTIONS:
• Retry analysis when service is available
================================================================================
BOOK 4: "Christmas 101"
------------------------------------------------------------
Overall Score: 70%
SECTION SCORES:
Author Information: 70% - Analysis unavailable - API error
Publishing Date: 70% - Analysis unavailable - API error
Why Match Explanations: 70% - Analysis unavailable - API error
Relevance Scores: 70% - Analysis unavailable - API error
DETAILED FEEDBACK:
• AI analysis temporarily unavailable due to service overload
IMPROVEMENT SUGGESTIONS:
• Retry analysis when service is available
================================================================================
BOOK 5: "So This Is Christmas"
------------------------------------------------------------
Overall Score: 70%
SECTION SCORES:
Author Information: 70% - Analysis unavailable - API error
Publishing Date: 70% - Analysis unavailable - API error
Why Match Explanations: 70% - Analysis unavailable - API error
Relevance Scores: 70% - Analysis unavailable - API error
DETAILED FEEDBACK:
• AI analysis temporarily unavailable due to service overload
IMPROVEMENT SUGGESTIONS:
• Retry analysis when service is available
================================================================================
SUMMARY FEEDBACK:
----------------------------------------
• AI analysis service is temporarily overloaded
• Using fallback analysis with neutral scores
• Error: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-pro:generateContent: [429 Too Many Requests] You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits.
* Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests, limit: 2
Please retry in 15.479670081s. [{"@type":"type.googleapis.com/google.rpc.QuotaFailure","violations":[{"quotaMetric":"generativelanguage.googleapis.com/generate_content_free_tier_requests","quotaId":"GenerateRequestsPerMinutePerProjectPerModel-FreeTier","quotaDimensions":{"location":"global","model":"gemini-2.5-pro"},"quotaValue":"2"}]},{"@type":"type.googleapis.com/google.rpc.Help","links":[{"description":"Learn more about Gemini API quotas","url":"https://ai.google.dev/gemini-api/docs/rate-limits"}]},{"@type":"type.googleapis.com/google.rpc.RetryInfo","retryDelay":"15s"}]
OVERALL IMPROVEMENT SUGGESTIONS:
----------------------------------------
• Retry the analysis when Gemini API is less busy
• Consider running tests during off-peak hours
• Check Google AI status page for service updates
🤖 PER-BOOK AI Response Relevance Validation
Query: "Suggest 5 books on christmas"
Overall Relevance Score: 70%
Status: FAIL
Books Analyzed: 5
Individual Book Analyses
📖 Book 1: "Finding Christmas" - 70%
Author Information: 70%
Analysis unavailable - API error
Publishing Date: 70%
Analysis unavailable - API error
Why Match Explanations: 70%
Analysis unavailable - API error
Relevance Scores: 70%
Analysis unavailable - API error
Improvement Suggestions:
💡 Retry analysis when service is available
📖 Book 2: "How the Dukes Stole Christmas" - 70%
Author Information: 70%
Analysis unavailable - API error
Publishing Date: 70%
Analysis unavailable - API error
Why Match Explanations: 70%
Analysis unavailable - API error
Relevance Scores: 70%
Analysis unavailable - API error
Improvement Suggestions:
💡 Retry analysis when service is available
📖 Book 3: "The First Christmas" - 70%
Author Information: 70%
Analysis unavailable - API error
Publishing Date: 70%
Analysis unavailable - API error
Why Match Explanations: 70%
Analysis unavailable - API error
Relevance Scores: 70%
Analysis unavailable - API error
Improvement Suggestions:
💡 Retry analysis when service is available
📖 Book 4: "Christmas 101" - 70%
Author Information: 70%
Analysis unavailable - API error
Publishing Date: 70%
Analysis unavailable - API error
Why Match Explanations: 70%
Analysis unavailable - API error
Relevance Scores: 70%
Analysis unavailable - API error
Improvement Suggestions:
💡 Retry analysis when service is available
📖 Book 5: "So This Is Christmas" - 70%
Author Information: 70%
Analysis unavailable - API error
Publishing Date: 70%
Analysis unavailable - API error
Why Match Explanations: 70%
Analysis unavailable - API error
Relevance Scores: 70%
Analysis unavailable - API error
Improvement Suggestions:
💡 Retry analysis when service is available
Summary Feedback
• AI analysis service is temporarily overloaded
• Using fallback analysis with neutral scores
• Error: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-pro:generateContent: [429 Too Many Requests] You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits. * Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests, limit: 2 Please retry in 15.479670081s. [{"@type":"type.googleapis.com/google.rpc.QuotaFailure","violations":[{"quotaMetric":"generativelanguage.googleapis.com/generate_content_free_tier_requests","quotaId":"GenerateRequestsPerMinutePerProjectPerModel-FreeTier","quotaDimensions":{"location":"global","model":"gemini-2.5-pro"},"quotaValue":"2"}]},{"@type":"type.googleapis.com/google.rpc.Help","links":[{"description":"Learn more about Gemini API quotas","url":"https://ai.google.dev/gemini-api/docs/rate-limits"}]},{"@type":"type.googleapis.com/google.rpc.RetryInfo","retryDelay":"15s"}]
Overall Improvement Suggestions
💡 Retry the analysis when Gemini API is less busy
💡 Consider running tests during off-peak hours
💡 Check Google AI status page for service updates
=== SCENARIO LOGS: Book Genie Response Validation with Citation Verification ===
[6:14:27 PM] Navigating to Creative Workspace login page
[6:14:30 PM] Homepage visibility: false
[6:14:30 PM] ℹ️ Typing query: "Suggest 5 books on christmas"
[6:14:30 PM] ℹ️ Preparing to type query: "Suggest 5 books on christmas"
[6:14:30 PM] ℹ️ Waiting for chat input to be visible
[6:14:31 PM] ℹ️ Clearing chat input
[6:14:31 PM] ℹ️ Typing query: "Suggest 5 books on christmas"
[6:14:33 PM] ℹ️ Pressing Enter to submit query
[6:14:33 PM] ✅ Query submitted successfully: "Suggest 5 books on christmas"
[6:14:33 PM] ℹ️ Waiting for AI to complete thinking...
[6:14:33 PM] 📋 WAITING FOR AI RESPONSE
[6:14:33 PM] ℹ️ Looking for AI thinking indicator...
[6:14:33 PM] ℹ️ Waiting for thinking indicator to appear (max 2 minutes)
[6:14:35 PM] ✅ ✓ AI thinking indicator appeared - AI is processing the request
[6:14:35 PM] ℹ️ Waiting for thinking indicator to disappear (max 5 minutes)
[6:14:38 PM] ✅ ✓ AI thinking completed - Response should be ready
[6:14:38 PM] ℹ️ 🔍 Checking for "None of the above" option...
[6:14:38 PM] ℹ️ ✓ "None of the above" option found - preparing to click
[6:14:38 PM] ✅ ✅ Clicked on "None of the above" option
[6:14:38 PM] ⚠ "None of the above" option still visible after click
[6:14:38 PM] ℹ️ ⏳ Waiting for AI to process "None of the above" selection...
[6:14:39 PM] ℹ️ ✓ AI thinking indicator appeared after "None of the above" selection
[6:15:11 PM] ✅ ✅ AI thinking completed after "None of the above" selection
[6:15:13 PM] ✅ ✅ "None of the above" handling completed successfully
[6:15:13 PM] 📊 "None of the above" option was handled successfully
[6:15:13 PM] ℹ️ Allowing additional time for response rendering
[6:15:16 PM] ✅ AI response wait process completed
[6:15:16 PM] ℹ️ Validating response for query: "Suggest 5 books on christmas"
[6:15:16 PM] 📋 VALIDATING RESPONSE FOR QUERY: "Suggest 5 books on christmas"
[6:15:16 PM] ℹ️ Locating the latest response in chat
[6:15:16 PM] ℹ️ Waiting for response to be visible (max 30 seconds)
[6:15:16 PM] ✅ Response received - Length: 2923 characters
[6:15:16 PM] ℹ️ Response preview: Watch Me WorkInterpreting context for your query...Retrieving Relevant Books for 'books with Christm...
[6:15:16 PM] ℹ️ Extracting book data from response...
[6:15:16 PM] 📋 EXTRACTING BOOK DATA FROM RESPONSE
[6:15:16 PM] ℹ️ Locating BookGenie response container
[6:15:16 PM] ℹ️ Extracting HTML content from response
[6:15:16 PM] ℹ️ HTML content extracted - Length: 34874 characters
[6:15:16 PM] ℹ️ Parsing HTML to extract book data
[6:15:16 PM] ✅ SUCCESSFULLY EXTRACTED 5 BOOKS
[6:15:16 PM] 📋 EXTRACTED BOOKS SUMMARY
[6:15:16 PM] 📚 1. "Finding Christmas" - Score: 100%%
[6:15:16 PM] 📚 2. "How the Dukes Stole Christmas" - Score: 100%%
[6:15:16 PM] 📚 3. "The First Christmas" - Score: 100%%
[6:15:16 PM] 📚 4. "Christmas 101" - Score: 100%%
[6:15:16 PM] 📚 5. "So This Is Christmas" - Score: 100%%
[6:15:16 PM] 📊 Total books extracted: 5
[6:15:16 PM] 📊 Average relevance score: 100%
[6:15:16 PM] ✅ Extracted 5 books from response
[6:15:16 PM] ℹ️ Saving 5 books to Excel file: test_results/suggest_5_books_on_christmas.xlsx
[6:15:16 PM] 📋 SAVING BOOKS TO EXCEL FILE
[6:15:16 PM] ℹ️ Target file path: test_results/suggest_5_books_on_christmas.xlsx
[6:15:16 PM] ℹ️ Exporting 5 books to Excel format
[6:15:16 PM] ✅ Book data saved to: test_results/suggest_5_books_on_christmas.xlsx
[6:15:16 PM] 📊 File verified - Size: 17.49 KB
[6:15:16 PM] ✅ ✓ Excel file creation verified successfully
[6:15:17 PM] 📋 STARTING INDIVIDUAL BOOK VALIDATION (5 books) for query: "Suggest 5 books on christmas"
[6:15:17 PM] 📋 📚 VALIDATING BOOK 1: "Finding Christmas"
[6:15:17 PM] ℹ️ Starting comprehensive book validation
[6:15:17 PM] ℹ️ Validating title: "Finding Christmas" vs expected: "Finding Christmas"
[6:15:17 PM] ✅ ✅ TITLE VALIDATION PASSED: "Finding Christmas"
[6:15:17 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:15:17 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:15:17 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:15:17 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:15:17 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:15:17 PM] ✅ 🎉 BOOK 1 VALIDATION: ALL CHECKS PASSED!
[6:15:17 PM] ℹ️ Book 1 validation completed
[6:15:17 PM] 📋 📚 VALIDATING BOOK 2: "How the Dukes Stole Christmas"
[6:15:17 PM] ℹ️ Starting comprehensive book validation
[6:15:17 PM] ℹ️ Validating title: "How the Dukes Stole Christmas" vs expected: "How the Dukes Stole Christmas"
[6:15:17 PM] ✅ ✅ TITLE VALIDATION PASSED: "How the Dukes Stole Christmas"
[6:15:17 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:15:17 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:15:17 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:15:17 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:15:17 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:15:17 PM] ✅ 🎉 BOOK 2 VALIDATION: ALL CHECKS PASSED!
[6:15:17 PM] ℹ️ Book 2 validation completed
[6:15:17 PM] 📋 📚 VALIDATING BOOK 3: "The First Christmas"
[6:15:17 PM] ℹ️ Starting comprehensive book validation
[6:15:17 PM] ℹ️ Validating title: "The First Christmas" vs expected: "The First Christmas"
[6:15:17 PM] ✅ ✅ TITLE VALIDATION PASSED: "The First Christmas"
[6:15:17 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:15:17 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:15:17 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:15:17 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:15:17 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:15:17 PM] ✅ 🎉 BOOK 3 VALIDATION: ALL CHECKS PASSED!
[6:15:17 PM] ℹ️ Book 3 validation completed
[6:15:17 PM] 📋 📚 VALIDATING BOOK 4: "Christmas 101"
[6:15:17 PM] ℹ️ Starting comprehensive book validation
[6:15:17 PM] ℹ️ Validating title: "Christmas 101" vs expected: "Christmas 101"
[6:15:17 PM] ✅ ✅ TITLE VALIDATION PASSED: "Christmas 101"
[6:15:17 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:15:17 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:15:17 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:15:17 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:15:17 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:15:17 PM] ✅ 🎉 BOOK 4 VALIDATION: ALL CHECKS PASSED!
[6:15:17 PM] ℹ️ Book 4 validation completed
[6:15:17 PM] 📋 📚 VALIDATING BOOK 5: "So This Is Christmas"
[6:15:17 PM] ℹ️ Starting comprehensive book validation
[6:15:17 PM] ℹ️ Validating title: "So This Is Christmas" vs expected: "So This Is Christmas"
[6:15:17 PM] ✅ ✅ TITLE VALIDATION PASSED: "So This Is Christmas"
[6:15:17 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:15:17 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:15:17 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:15:17 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:15:17 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:15:17 PM] ✅ 🎉 BOOK 5 VALIDATION: ALL CHECKS PASSED!
[6:15:17 PM] ℹ️ Book 5 validation completed
[6:15:17 PM] 📊 VALIDATION SUMMARY: 5/5 books passed
[6:15:17 PM] ✅ ALL BOOKS VALIDATED SUCCESSFULLY!
[6:15:17 PM] 📋 🔍 VALIDATING REASON-CITATION MATCHES
[6:15:17 PM] ℹ️ Starting citation validation process...
[6:15:17 PM] 📋 🚀 STARTING COMPLETE CITATION VALIDATION WORKFLOW
[6:15:17 PM] ℹ️ Step 1: Extracting books from AI response
[6:15:17 PM] 📋 EXTRACTING BOOK DATA FROM RESPONSE
[6:15:17 PM] ℹ️ Locating BookGenie response container
[6:15:17 PM] ℹ️ Extracting HTML content from response
[6:15:17 PM] ℹ️ HTML content extracted - Length: 34874 characters
[6:15:17 PM] ℹ️ Parsing HTML to extract book data
[6:15:17 PM] ✅ SUCCESSFULLY EXTRACTED 5 BOOKS
[6:15:17 PM] 📋 EXTRACTED BOOKS SUMMARY
[6:15:17 PM] 📚 1. "Finding Christmas" - Score: 100%%
[6:15:17 PM] 📚 2. "How the Dukes Stole Christmas" - Score: 100%%
[6:15:17 PM] 📚 3. "The First Christmas" - Score: 100%%
[6:15:17 PM] 📚 4. "Christmas 101" - Score: 100%%
[6:15:17 PM] 📚 5. "So This Is Christmas" - Score: 100%%
[6:15:17 PM] 📊 Total books extracted: 5
[6:15:17 PM] 📊 Average relevance score: 100%
[6:15:17 PM] ℹ️ Step 2: Extracting citation texts from book elements
[6:15:17 PM] 📋 📚 EXTRACTING CITATION TEXTS FROM BOOK ELEMENTS
[6:15:17 PM] ℹ️ Step 1: Waiting for main book section
[6:15:17 PM] ℹ️ Found 11 total accordion sections
[6:15:17 PM] ℹ️ Found individual book: "1. Finding Christmas"
[6:15:17 PM] ℹ️ Found individual book: "2. How the Dukes Stole Christmas"
[6:15:17 PM] ℹ️ Found individual book: "3. The First Christmas"
[6:15:17 PM] ℹ️ Found individual book: "4. Christmas 101"
[6:15:17 PM] ℹ️ Found individual book: "5. So This Is Christmas"
[6:15:17 PM] ℹ️ Processing 5 individual book sections
[6:15:17 PM] 📋 📖 PROCESSING BOOK 1: "Finding Christmas"
[6:15:17 PM] ℹ️ Expanding book section: "Finding Christmas"
[6:15:20 PM] ✅ Book section expanded: "Finding Christmas"
[6:15:20 PM] ℹ️ Expanding "Why this book is the match" section for: "Finding Christmas"
[6:15:23 PM] ✅ "Why this book is the match" section expanded for: "Finding Christmas"
[6:15:23 PM] ℹ️ Found 3 citation buttons for "Finding Christmas"
[6:15:23 PM] ℹ️ Extracting citation 1/3 for "Finding Christmas"
[6:15:23 PM] ℹ️ Processing metadata citation 1 for "Finding Christmas"
[6:15:24 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:15:26 PM] ℹ️ Extracted citation text (first 100 chars): Finding Christmas: A Novel...
[6:15:26 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:15:27 PM] ✅ ✅ metadata citation 1 closed successfully
[6:15:27 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 26)
[6:15:27 PM] ℹ️ Extracting citation 2/3 for "Finding Christmas"
[6:15:27 PM] ℹ️ Processing metadata citation 2 for "Finding Christmas"
[6:15:28 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:15:30 PM] ℹ️ Extracted citation text (first 100 chars): christmas books...
[6:15:30 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:15:32 PM] ✅ ✅ metadata citation 2 closed successfully
[6:15:32 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 15)
[6:15:32 PM] ℹ️ Extracting citation 3/3 for "Finding Christmas"
[6:15:32 PM] ℹ️ Processing metadata citation 3 for "Finding Christmas"
[6:15:33 PM] ℹ️ Clicking to OPEN metadata citation 3
[6:15:35 PM] ℹ️ Extracted citation text (first 100 chars): comes a heartwarming new Christmas story...
[6:15:35 PM] ℹ️ Clicking to CLOSE metadata citation 3
[6:15:37 PM] ✅ ✅ metadata citation 3 closed successfully
[6:15:37 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 40)
[6:15:37 PM] ℹ️ Collapsing book section: "Finding Christmas"
[6:15:38 PM] ✅ Book section collapsed: "Finding Christmas"
[6:15:38 PM] ✅ ✅ Completed processing "Finding Christmas" - 3 citations extracted
[6:15:38 PM] 📋 📖 PROCESSING BOOK 2: "How the Dukes Stole Christmas"
[6:15:38 PM] ℹ️ Expanding book section: "How the Dukes Stole Christmas"
[6:15:41 PM] ✅ Book section expanded: "How the Dukes Stole Christmas"
[6:15:41 PM] ℹ️ Expanding "Why this book is the match" section for: "How the Dukes Stole Christmas"
[6:15:44 PM] ✅ "Why this book is the match" section expanded for: "How the Dukes Stole Christmas"
[6:15:44 PM] ℹ️ Found 3 citation buttons for "How the Dukes Stole Christmas"
[6:15:44 PM] ℹ️ Extracting citation 1/3 for "How the Dukes Stole Christmas"
[6:15:44 PM] ℹ️ Processing metadata citation 1 for "How the Dukes Stole Christmas"
[6:15:45 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:15:48 PM] ℹ️ Extracted citation text (first 100 chars): Book Title: How the Dukes Stole Christmas: A Christmas Romance Anthology...
[6:15:48 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:15:49 PM] ✅ ✅ metadata citation 1 closed successfully
[6:15:49 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 72)
[6:15:49 PM] ℹ️ Extracting citation 2/3 for "How the Dukes Stole Christmas"
[6:15:49 PM] ℹ️ Processing metadata citation 2 for "How the Dukes Stole Christmas"
[6:15:50 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:15:52 PM] ℹ️ Extracted citation text (first 100 chars): christmas books...
[6:15:52 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:15:54 PM] ✅ ✅ metadata citation 2 closed successfully
[6:15:54 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 15)
[6:15:54 PM] ℹ️ Extracting citation 3/3 for "How the Dukes Stole Christmas"
[6:15:54 PM] ℹ️ Processing metadata citation 3 for "How the Dukes Stole Christmas"
[6:15:55 PM] ℹ️ Clicking to OPEN metadata citation 3
[6:15:57 PM] ℹ️ Extracted citation text (first 100 chars): Christmas Romance Anthology...
[6:15:57 PM] ℹ️ Clicking to CLOSE metadata citation 3
[6:15:58 PM] ✅ ✅ metadata citation 3 closed successfully
[6:15:58 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 27)
[6:15:58 PM] ℹ️ Collapsing book section: "How the Dukes Stole Christmas"
[6:16:00 PM] ✅ Book section collapsed: "How the Dukes Stole Christmas"
[6:16:00 PM] ✅ ✅ Completed processing "How the Dukes Stole Christmas" - 3 citations extracted
[6:16:00 PM] 📋 📖 PROCESSING BOOK 3: "The First Christmas"
[6:16:00 PM] ℹ️ Expanding book section: "The First Christmas"
[6:16:03 PM] ✅ Book section expanded: "The First Christmas"
[6:16:03 PM] ℹ️ Expanding "Why this book is the match" section for: "The First Christmas"
[6:16:06 PM] ✅ "Why this book is the match" section expanded for: "The First Christmas"
[6:16:06 PM] ℹ️ Found 3 citation buttons for "The First Christmas"
[6:16:06 PM] ℹ️ Extracting citation 1/3 for "The First Christmas"
[6:16:06 PM] ℹ️ Processing metadata citation 1 for "The First Christmas"
[6:16:07 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:16:09 PM] ℹ️ Extracted citation text (first 100 chars): Book Title: The First Christmas...
[6:16:09 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:16:11 PM] ✅ ✅ metadata citation 1 closed successfully
[6:16:11 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 31)
[6:16:11 PM] ℹ️ Extracting citation 2/3 for "The First Christmas"
[6:16:11 PM] ℹ️ Processing metadata citation 2 for "The First Christmas"
[6:16:12 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:16:14 PM] ℹ️ Extracted citation text (first 100 chars): christmas books...
[6:16:14 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:16:15 PM] ✅ ✅ metadata citation 2 closed successfully
[6:16:15 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 15)
[6:16:15 PM] ℹ️ Extracting citation 3/3 for "The First Christmas"
[6:16:15 PM] ℹ️ Processing manuscript citation 3 for "The First Christmas"
[6:16:16 PM] ℹ️ Clicking to OPEN manuscript citation 3
[6:16:18 PM] ℹ️ Extracted citation text (first 100 chars): The idea to write this book about the stories of the first Christmas...
[6:16:18 PM] ℹ️ Clicking to CLOSE manuscript citation 3
[6:16:20 PM] ✅ ✅ manuscript citation 3 closed successfully
[6:16:20 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 68)
[6:16:20 PM] ℹ️ Collapsing book section: "The First Christmas"
[6:16:21 PM] ✅ Book section collapsed: "The First Christmas"
[6:16:21 PM] ✅ ✅ Completed processing "The First Christmas" - 3 citations extracted
[6:16:21 PM] 📋 📖 PROCESSING BOOK 4: "Christmas 101"
[6:16:21 PM] ℹ️ Expanding book section: "Christmas 101"
[6:16:25 PM] ✅ Book section expanded: "Christmas 101"
[6:16:25 PM] ℹ️ Expanding "Why this book is the match" section for: "Christmas 101"
[6:16:28 PM] ✅ "Why this book is the match" section expanded for: "Christmas 101"
[6:16:28 PM] ℹ️ Found 3 citation buttons for "Christmas 101"
[6:16:28 PM] ℹ️ Extracting citation 1/3 for "Christmas 101"
[6:16:28 PM] ℹ️ Processing metadata citation 1 for "Christmas 101"
[6:16:29 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:16:31 PM] ℹ️ Extracted citation text (first 100 chars): Book Title: Christmas 101...
[6:16:31 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:16:32 PM] ✅ ✅ metadata citation 1 closed successfully
[6:16:32 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 25)
[6:16:32 PM] ℹ️ Extracting citation 2/3 for "Christmas 101"
[6:16:32 PM] ℹ️ Processing metadata citation 2 for "Christmas 101"
[6:16:33 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:16:35 PM] ℹ️ Extracted citation text (first 100 chars): christmas books...
[6:16:35 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:16:37 PM] ✅ ✅ metadata citation 2 closed successfully
[6:16:37 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 15)
[6:16:37 PM] ℹ️ Extracting citation 3/3 for "Christmas 101"
[6:16:37 PM] ℹ️ Processing metadata citation 3 for "Christmas 101"
[6:16:38 PM] ℹ️ Clicking to OPEN metadata citation 3
[6:16:40 PM] ℹ️ Extracted citation text (first 100 chars): holiday planning simple, fun and festive...
[6:16:40 PM] ℹ️ Clicking to CLOSE metadata citation 3
[6:16:41 PM] ✅ ✅ metadata citation 3 closed successfully
[6:16:41 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 40)
[6:16:41 PM] ℹ️ Collapsing book section: "Christmas 101"
[6:16:43 PM] ✅ Book section collapsed: "Christmas 101"
[6:16:43 PM] ✅ ✅ Completed processing "Christmas 101" - 3 citations extracted
[6:16:43 PM] 📋 📖 PROCESSING BOOK 5: "So This Is Christmas"
[6:16:43 PM] ℹ️ Expanding book section: "So This Is Christmas"
[6:16:46 PM] ✅ Book section expanded: "So This Is Christmas"
[6:16:46 PM] ℹ️ Expanding "Why this book is the match" section for: "So This Is Christmas"
[6:16:49 PM] ✅ "Why this book is the match" section expanded for: "So This Is Christmas"
[6:16:49 PM] ℹ️ Found 3 citation buttons for "So This Is Christmas"
[6:16:49 PM] ℹ️ Extracting citation 1/3 for "So This Is Christmas"
[6:16:49 PM] ℹ️ Processing metadata citation 1 for "So This Is Christmas"
[6:16:50 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:16:52 PM] ℹ️ Extracted citation text (first 100 chars): Book Title: So This Is Christmas...
[6:16:52 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:16:54 PM] ✅ ✅ metadata citation 1 closed successfully
[6:16:54 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 32)
[6:16:54 PM] ℹ️ Extracting citation 2/3 for "So This Is Christmas"
[6:16:54 PM] ℹ️ Processing metadata citation 2 for "So This Is Christmas"
[6:16:55 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:16:57 PM] ℹ️ Extracted citation text (first 100 chars): christmas books...
[6:16:57 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:16:58 PM] ✅ ✅ metadata citation 2 closed successfully
[6:16:58 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 15)
[6:16:58 PM] ℹ️ Extracting citation 3/3 for "So This Is Christmas"
[6:16:58 PM] ℹ️ Processing metadata citation 3 for "So This Is Christmas"
[6:16:59 PM] ℹ️ Clicking to OPEN metadata citation 3
[6:17:01 PM] ℹ️ Extracted citation text (first 100 chars): the country’s magical Christmas season...
[6:17:01 PM] ℹ️ Clicking to CLOSE metadata citation 3
[6:17:03 PM] ✅ ✅ metadata citation 3 closed successfully
[6:17:03 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 38)
[6:17:03 PM] ℹ️ Collapsing book section: "So This Is Christmas"
[6:17:05 PM] ✅ Book section collapsed: "So This Is Christmas"
[6:17:05 PM] ✅ ✅ Completed processing "So This Is Christmas" - 3 citations extracted
[6:17:05 PM] ✅ 🎉 Successfully extracted citations from 5 books
[6:17:05 PM] ℹ️ Step 3: Validating reason-citation matches
[6:17:05 PM] 📋 🔍 VALIDATING REASON-CITATION MATCHES (80% THRESHOLD)
[6:17:05 PM] ℹ️ 📖 Validating 3 reasons for: "Finding Christmas"
[6:17:05 PM] ℹ️ Reason 1: Checking match...
[6:17:15 PM] ✅ ✅ "Finding Christmas" - Reason 1: 95% match - PASS
[6:17:15 PM] ℹ️ Reason 2: Checking match...
[6:17:15 PM] ✅ ✅ "Finding Christmas" - Reason 2: 100% match - PASS
[6:17:15 PM] ℹ️ Reason 3: Checking match...
[6:17:26 PM] ✅ ✅ "Finding Christmas" - Reason 3: 95% match - PASS
[6:17:26 PM] ℹ️ 📖 Validating 3 reasons for: "How the Dukes Stole Christmas"
[6:17:26 PM] ℹ️ Reason 1: Checking match...
[6:17:36 PM] ✅ ✅ "How the Dukes Stole Christmas" - Reason 1: 95% match - PASS
[6:17:36 PM] ℹ️ Reason 2: Checking match...
[6:17:36 PM] ✅ ✅ "How the Dukes Stole Christmas" - Reason 2: 100% match - PASS
[6:17:36 PM] ℹ️ Reason 3: Checking match...
[6:17:36 PM] ✅ ✅ "How the Dukes Stole Christmas" - Reason 3: 100% match - PASS
[6:17:36 PM] ℹ️ 📖 Validating 3 reasons for: "The First Christmas"
[6:17:36 PM] ℹ️ Reason 1: Checking match...
[6:17:36 PM] ❌ ❌ "The First Christmas" - Reason 1: 0% match - FAIL
[6:17:36 PM] ❌ ⚠️ Citation text not found in reason: "Book Title: The First Christmas"
[6:17:36 PM] ℹ️ Reason 2: Checking match...
[6:17:36 PM] ✅ ✅ "The First Christmas" - Reason 2: 100% match - PASS
[6:17:36 PM] ℹ️ Reason 3: Checking match...
[6:17:36 PM] ❌ ❌ "The First Christmas" - Reason 3: 0% match - FAIL
[6:17:36 PM] ❌ ⚠️ Citation text not found in reason: "The idea to write this book about the stories of the first Christmas"
[6:17:36 PM] ℹ️ 📖 Validating 3 reasons for: "Christmas 101"
[6:17:36 PM] ℹ️ Reason 1: Checking match...
[6:17:36 PM] ❌ ❌ "Christmas 101" - Reason 1: 0% match - FAIL
[6:17:36 PM] ❌ ⚠️ Citation text not found in reason: "Book Title: Christmas 101"
[6:17:36 PM] ℹ️ Reason 2: Checking match...
[6:17:36 PM] ✅ ✅ "Christmas 101" - Reason 2: 100% match - PASS
[6:17:36 PM] ℹ️ Reason 3: Checking match...
[6:17:36 PM] ✅ ✅ "Christmas 101" - Reason 3: 100% match - PASS
[6:17:36 PM] ℹ️ 📖 Validating 3 reasons for: "So This Is Christmas"
[6:17:36 PM] ℹ️ Reason 1: Checking match...
[6:17:36 PM] ❌ ❌ "So This Is Christmas" - Reason 1: 0% match - FAIL
[6:17:36 PM] ❌ ⚠️ Citation text not found in reason: "Book Title: So This Is Christmas"
[6:17:36 PM] ℹ️ Reason 2: Checking match...
[6:17:36 PM] ✅ ✅ "So This Is Christmas" - Reason 2: 100% match - PASS
[6:17:36 PM] ℹ️ Reason 3: Checking match...
[6:17:36 PM] ✅ ✅ "So This Is Christmas" - Reason 3: 100% match - PASS
[6:17:36 PM] 📊 📊 CITATION VALIDATION SUMMARY: 11/15 passed (73.3%)
[6:17:36 PM] ❌ 💥 REASON-CITATION VALIDATION: OVERALL FAILED
[6:17:36 PM] ℹ️ Step 4: Generating detailed report
[6:17:36 PM] 📋 📋 GENERATING CITATION VALIDATION REPORT
[6:17:36 PM] ℹ️ Report generated: 11/15 reasons passed
[6:17:36 PM] ✅ ✅ Citation validation workflow completed successfully
[6:17:36 PM] 📊 📈 OVERALL CITATION VALIDATION: 11/15 (73.3%)
[6:17:36 PM] ❌ ❌ REASON-CITATION VALIDATION: OVERALL FAILED - Only 73.3% of reasons match citations (required: 80%)
[6:17:36 PM] 📋 📋 GENERATING DETAILED CITATION VALIDATION REPORT
[6:17:36 PM] ℹ️ Generating HTML citation validation report
[6:17:36 PM] ✅ Generated HTML report for 15 reasons
[6:17:36 PM] ✅ ✅ Citation validation HTML report generated and attached
[6:17:36 PM] ✅ ✅ Citation validation plain text report attached
[6:17:36 PM] ℹ️ Checking database connectivity...
[6:17:37 PM] ✅ ✅ Database connected successfully
[6:17:37 PM] ℹ️ Database path: /home/runner/work/HC_Epub_Automation/HC_Epub_Automation/tests/test_data/database.xlsx
[6:17:37 PM] ℹ️ Total books: 10032
[6:17:37 PM] 📊 Database contains 10032 books
[6:17:37 PM] ✅ ✅ Database book count requirement met: 10032 >= 10032
[6:17:37 PM] 📋 🔍 VALIDATING EXTRACTED BOOKS INDIVIDUALLY AGAINST DATABASE
[6:17:37 PM] ℹ️ Checking book 1/5: "Finding Christmas"
[6:17:37 PM] ℹ️ Checking if book exists in database: "Finding Christmas"
[6:17:37 PM] ✅ ✅ Database match: "Finding Christmas" → "Finding Christmas"
[6:17:37 PM] ℹ️ Checking book 2/5: "How the Dukes Stole Christmas"
[6:17:37 PM] ℹ️ Checking if book exists in database: "How the Dukes Stole Christmas"
[6:17:37 PM] ✅ ✅ Database match: "How the Dukes Stole Christmas" → "How the Dukes Stole Christmas"
[6:17:37 PM] ℹ️ Checking book 3/5: "The First Christmas"
[6:17:37 PM] ℹ️ Checking if book exists in database: "The First Christmas"
[6:17:38 PM] ✅ ✅ Database match: "The First Christmas" → "The First Christmas"
[6:17:38 PM] ℹ️ Checking book 4/5: "Christmas 101"
[6:17:38 PM] ℹ️ Checking if book exists in database: "Christmas 101"
[6:17:38 PM] ✅ ✅ Database match: "Christmas 101" → "Christmas 101"
[6:17:38 PM] ℹ️ Checking book 5/5: "So This Is Christmas"
[6:17:38 PM] ℹ️ Checking if book exists in database: "So This Is Christmas"
[6:17:38 PM] ✅ ✅ Database match: "So This Is Christmas" → "So This Is Christmas"
[6:17:38 PM] 📊 INDIVIDUAL DATABASE VALIDATION SUMMARY: 5/5 books found in database
[6:17:38 PM] ✅ 🎉 ALL BOOKS VALIDATED SUCCESSFULLY AGAINST DATABASE!
[6:17:38 PM] 📋 🤖 VALIDATING RESPONSE RELEVANCE WITH GEMINI AI - PER BOOK ANALYSIS
[6:17:38 PM] ℹ️ Starting Gemini AI per-book relevance validation...
[6:17:38 PM] 📋 🤖 ANALYZING RESPONSE RELEVANCE WITH GEMINI AI - PER BOOK ANALYSIS
[6:17:38 PM] ℹ️ Starting Gemini AI analysis with retry logic...
[6:17:44 PM] ⚠ ⚠️ GEMINI VALIDATION: Using fallback analysis due to API issues
[6:17:44 PM] ℹ️ Overall Score: 70% (Fallback)
[6:17:44 PM] 📋 📖 ANALYSIS FOR: "Finding Christmas" - Score: 70%
[6:17:44 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ 💡 Book-specific improvements:
[6:17:44 PM] ℹ️ - Retry analysis when service is available
[6:17:44 PM] 📋 📖 ANALYSIS FOR: "How the Dukes Stole Christmas" - Score: 70%
[6:17:44 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ 💡 Book-specific improvements:
[6:17:44 PM] ℹ️ - Retry analysis when service is available
[6:17:44 PM] 📋 📖 ANALYSIS FOR: "The First Christmas" - Score: 70%
[6:17:44 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ 💡 Book-specific improvements:
[6:17:44 PM] ℹ️ - Retry analysis when service is available
[6:17:44 PM] 📋 📖 ANALYSIS FOR: "Christmas 101" - Score: 70%
[6:17:44 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ 💡 Book-specific improvements:
[6:17:44 PM] ℹ️ - Retry analysis when service is available
[6:17:44 PM] 📋 📖 ANALYSIS FOR: "So This Is Christmas" - Score: 70%
[6:17:44 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ 💡 Book-specific improvements:
[6:17:44 PM] ℹ️ - Retry analysis when service is available
[6:17:44 PM] ℹ️ 📊 SUMMARY FEEDBACK:
[6:17:44 PM] ℹ️ • AI analysis service is temporarily overloaded
[6:17:44 PM] ℹ️ • Using fallback analysis with neutral scores
[6:17:44 PM] ℹ️ • Error: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-pro:generateContent: [429 Too Many Requests] You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits.
* Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests, limit: 2
Please retry in 15.479670081s. [{"@type":"type.googleapis.com/google.rpc.QuotaFailure","violations":[{"quotaMetric":"generativelanguage.googleapis.com/generate_content_free_tier_requests","quotaId":"GenerateRequestsPerMinutePerProjectPerModel-FreeTier","quotaDimensions":{"location":"global","model":"gemini-2.5-pro"},"quotaValue":"2"}]},{"@type":"type.googleapis.com/google.rpc.Help","links":[{"description":"Learn more about Gemini API quotas","url":"https://ai.google.dev/gemini-api/docs/rate-limits"}]},{"@type":"type.googleapis.com/google.rpc.RetryInfo","retryDelay":"15s"}]
[6:17:44 PM] ❌ ❌ GEMINI AI VALIDATION FAILED: 70% relevance score (required: 80%)
[6:17:44 PM] ℹ️ Continuing execution due to soft assertion...
=== END LOGS ===
Scenario Logs: Book Genie Response Validation with Citation Verification
[6:14:27 PM] Navigating to Creative Workspace login page
[6:14:30 PM] Homepage visibility: false
[6:14:30 PM] ℹ️ Typing query: "Suggest 5 books on christmas"
[6:14:30 PM] ℹ️ Preparing to type query: "Suggest 5 books on christmas"
[6:14:30 PM] ℹ️ Waiting for chat input to be visible
[6:14:31 PM] ℹ️ Clearing chat input
[6:14:31 PM] ℹ️ Typing query: "Suggest 5 books on christmas"
[6:14:33 PM] ℹ️ Pressing Enter to submit query
[6:14:33 PM] ✅ Query submitted successfully: "Suggest 5 books on christmas"
[6:14:33 PM] ℹ️ Waiting for AI to complete thinking...
[6:14:33 PM] 📋 WAITING FOR AI RESPONSE
[6:14:33 PM] ℹ️ Looking for AI thinking indicator...
[6:14:33 PM] ℹ️ Waiting for thinking indicator to appear (max 2 minutes)
[6:14:35 PM] ✅ ✓ AI thinking indicator appeared - AI is processing the request
[6:14:35 PM] ℹ️ Waiting for thinking indicator to disappear (max 5 minutes)
[6:14:38 PM] ✅ ✓ AI thinking completed - Response should be ready
[6:14:38 PM] ℹ️ 🔍 Checking for "None of the above" option...
[6:14:38 PM] ℹ️ ✓ "None of the above" option found - preparing to click
[6:14:38 PM] ✅ ✅ Clicked on "None of the above" option
[6:14:38 PM] ⚠ "None of the above" option still visible after click
[6:14:38 PM] ℹ️ ⏳ Waiting for AI to process "None of the above" selection...
[6:14:39 PM] ℹ️ ✓ AI thinking indicator appeared after "None of the above" selection
[6:15:11 PM] ✅ ✅ AI thinking completed after "None of the above" selection
[6:15:13 PM] ✅ ✅ "None of the above" handling completed successfully
[6:15:13 PM] 📊 "None of the above" option was handled successfully
[6:15:13 PM] ℹ️ Allowing additional time for response rendering
[6:15:16 PM] ✅ AI response wait process completed
[6:15:16 PM] ℹ️ Validating response for query: "Suggest 5 books on christmas"
[6:15:16 PM] 📋 VALIDATING RESPONSE FOR QUERY: "Suggest 5 books on christmas"
[6:15:16 PM] ℹ️ Locating the latest response in chat
[6:15:16 PM] ℹ️ Waiting for response to be visible (max 30 seconds)
[6:15:16 PM] ✅ Response received - Length: 2923 characters
[6:15:16 PM] ℹ️ Response preview: Watch Me WorkInterpreting context for your query...Retrieving Relevant Books for 'books with Christm...
[6:15:16 PM] ℹ️ Extracting book data from response...
[6:15:16 PM] 📋 EXTRACTING BOOK DATA FROM RESPONSE
[6:15:16 PM] ℹ️ Locating BookGenie response container
[6:15:16 PM] ℹ️ Extracting HTML content from response
[6:15:16 PM] ℹ️ HTML content extracted - Length: 34874 characters
[6:15:16 PM] ℹ️ Parsing HTML to extract book data
[6:15:16 PM] ✅ SUCCESSFULLY EXTRACTED 5 BOOKS
[6:15:16 PM] 📋 EXTRACTED BOOKS SUMMARY
[6:15:16 PM] 📚 1. "Finding Christmas" - Score: 100%%
[6:15:16 PM] 📚 2. "How the Dukes Stole Christmas" - Score: 100%%
[6:15:16 PM] 📚 3. "The First Christmas" - Score: 100%%
[6:15:16 PM] 📚 4. "Christmas 101" - Score: 100%%
[6:15:16 PM] 📚 5. "So This Is Christmas" - Score: 100%%
[6:15:16 PM] 📊 Total books extracted: 5
[6:15:16 PM] 📊 Average relevance score: 100%
[6:15:16 PM] ✅ Extracted 5 books from response
[6:15:16 PM] ℹ️ Saving 5 books to Excel file: test_results/suggest_5_books_on_christmas.xlsx
[6:15:16 PM] 📋 SAVING BOOKS TO EXCEL FILE
[6:15:16 PM] ℹ️ Target file path: test_results/suggest_5_books_on_christmas.xlsx
[6:15:16 PM] ℹ️ Exporting 5 books to Excel format
[6:15:16 PM] ✅ Book data saved to: test_results/suggest_5_books_on_christmas.xlsx
[6:15:16 PM] 📊 File verified - Size: 17.49 KB
[6:15:16 PM] ✅ ✓ Excel file creation verified successfully
[6:15:17 PM] 📋 STARTING INDIVIDUAL BOOK VALIDATION (5 books) for query: "Suggest 5 books on christmas"
[6:15:17 PM] 📋 📚 VALIDATING BOOK 1: "Finding Christmas"
[6:15:17 PM] ℹ️ Starting comprehensive book validation
[6:15:17 PM] ℹ️ Validating title: "Finding Christmas" vs expected: "Finding Christmas"
[6:15:17 PM] ✅ ✅ TITLE VALIDATION PASSED: "Finding Christmas"
[6:15:17 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:15:17 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:15:17 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:15:17 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:15:17 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:15:17 PM] ✅ 🎉 BOOK 1 VALIDATION: ALL CHECKS PASSED!
[6:15:17 PM] ℹ️ Book 1 validation completed
[6:15:17 PM] 📋 📚 VALIDATING BOOK 2: "How the Dukes Stole Christmas"
[6:15:17 PM] ℹ️ Starting comprehensive book validation
[6:15:17 PM] ℹ️ Validating title: "How the Dukes Stole Christmas" vs expected: "How the Dukes Stole Christmas"
[6:15:17 PM] ✅ ✅ TITLE VALIDATION PASSED: "How the Dukes Stole Christmas"
[6:15:17 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:15:17 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:15:17 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:15:17 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:15:17 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:15:17 PM] ✅ 🎉 BOOK 2 VALIDATION: ALL CHECKS PASSED!
[6:15:17 PM] ℹ️ Book 2 validation completed
[6:15:17 PM] 📋 📚 VALIDATING BOOK 3: "The First Christmas"
[6:15:17 PM] ℹ️ Starting comprehensive book validation
[6:15:17 PM] ℹ️ Validating title: "The First Christmas" vs expected: "The First Christmas"
[6:15:17 PM] ✅ ✅ TITLE VALIDATION PASSED: "The First Christmas"
[6:15:17 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:15:17 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:15:17 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:15:17 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:15:17 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:15:17 PM] ✅ 🎉 BOOK 3 VALIDATION: ALL CHECKS PASSED!
[6:15:17 PM] ℹ️ Book 3 validation completed
[6:15:17 PM] 📋 📚 VALIDATING BOOK 4: "Christmas 101"
[6:15:17 PM] ℹ️ Starting comprehensive book validation
[6:15:17 PM] ℹ️ Validating title: "Christmas 101" vs expected: "Christmas 101"
[6:15:17 PM] ✅ ✅ TITLE VALIDATION PASSED: "Christmas 101"
[6:15:17 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:15:17 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:15:17 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:15:17 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:15:17 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:15:17 PM] ✅ 🎉 BOOK 4 VALIDATION: ALL CHECKS PASSED!
[6:15:17 PM] ℹ️ Book 4 validation completed
[6:15:17 PM] 📋 📚 VALIDATING BOOK 5: "So This Is Christmas"
[6:15:17 PM] ℹ️ Starting comprehensive book validation
[6:15:17 PM] ℹ️ Validating title: "So This Is Christmas" vs expected: "So This Is Christmas"
[6:15:17 PM] ✅ ✅ TITLE VALIDATION PASSED: "So This Is Christmas"
[6:15:17 PM] ℹ️ Validating score: 100% vs expected: 100%
[6:15:17 PM] ✅ ✅ SCORE VALIDATION PASSED: 100% (expected 100%)
[6:15:17 PM] ✅ ✅ PERFECT SCORE GAP VALIDATION: Book has 100% score and no gap mentioned
[6:15:17 PM] ℹ️ Why-match points: extracted 3 vs expected 3
[6:15:17 PM] ✅ ✅ WHY-MATCH VALIDATION PASSED: 3 points (expected 3)
[6:15:17 PM] ✅ 🎉 BOOK 5 VALIDATION: ALL CHECKS PASSED!
[6:15:17 PM] ℹ️ Book 5 validation completed
[6:15:17 PM] 📊 VALIDATION SUMMARY: 5/5 books passed
[6:15:17 PM] ✅ ALL BOOKS VALIDATED SUCCESSFULLY!
[6:15:17 PM] 📋 🔍 VALIDATING REASON-CITATION MATCHES
[6:15:17 PM] ℹ️ Starting citation validation process...
[6:15:17 PM] 📋 🚀 STARTING COMPLETE CITATION VALIDATION WORKFLOW
[6:15:17 PM] ℹ️ Step 1: Extracting books from AI response
[6:15:17 PM] 📋 EXTRACTING BOOK DATA FROM RESPONSE
[6:15:17 PM] ℹ️ Locating BookGenie response container
[6:15:17 PM] ℹ️ Extracting HTML content from response
[6:15:17 PM] ℹ️ HTML content extracted - Length: 34874 characters
[6:15:17 PM] ℹ️ Parsing HTML to extract book data
[6:15:17 PM] ✅ SUCCESSFULLY EXTRACTED 5 BOOKS
[6:15:17 PM] 📋 EXTRACTED BOOKS SUMMARY
[6:15:17 PM] 📚 1. "Finding Christmas" - Score: 100%%
[6:15:17 PM] 📚 2. "How the Dukes Stole Christmas" - Score: 100%%
[6:15:17 PM] 📚 3. "The First Christmas" - Score: 100%%
[6:15:17 PM] 📚 4. "Christmas 101" - Score: 100%%
[6:15:17 PM] 📚 5. "So This Is Christmas" - Score: 100%%
[6:15:17 PM] 📊 Total books extracted: 5
[6:15:17 PM] 📊 Average relevance score: 100%
[6:15:17 PM] ℹ️ Step 2: Extracting citation texts from book elements
[6:15:17 PM] 📋 📚 EXTRACTING CITATION TEXTS FROM BOOK ELEMENTS
[6:15:17 PM] ℹ️ Step 1: Waiting for main book section
[6:15:17 PM] ℹ️ Found 11 total accordion sections
[6:15:17 PM] ℹ️ Found individual book: "1. Finding Christmas"
[6:15:17 PM] ℹ️ Found individual book: "2. How the Dukes Stole Christmas"
[6:15:17 PM] ℹ️ Found individual book: "3. The First Christmas"
[6:15:17 PM] ℹ️ Found individual book: "4. Christmas 101"
[6:15:17 PM] ℹ️ Found individual book: "5. So This Is Christmas"
[6:15:17 PM] ℹ️ Processing 5 individual book sections
[6:15:17 PM] 📋 📖 PROCESSING BOOK 1: "Finding Christmas"
[6:15:17 PM] ℹ️ Expanding book section: "Finding Christmas"
[6:15:20 PM] ✅ Book section expanded: "Finding Christmas"
[6:15:20 PM] ℹ️ Expanding "Why this book is the match" section for: "Finding Christmas"
[6:15:23 PM] ✅ "Why this book is the match" section expanded for: "Finding Christmas"
[6:15:23 PM] ℹ️ Found 3 citation buttons for "Finding Christmas"
[6:15:23 PM] ℹ️ Extracting citation 1/3 for "Finding Christmas"
[6:15:23 PM] ℹ️ Processing metadata citation 1 for "Finding Christmas"
[6:15:24 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:15:26 PM] ℹ️ Extracted citation text (first 100 chars): Finding Christmas: A Novel...
[6:15:26 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:15:27 PM] ✅ ✅ metadata citation 1 closed successfully
[6:15:27 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 26)
[6:15:27 PM] ℹ️ Extracting citation 2/3 for "Finding Christmas"
[6:15:27 PM] ℹ️ Processing metadata citation 2 for "Finding Christmas"
[6:15:28 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:15:30 PM] ℹ️ Extracted citation text (first 100 chars): christmas books...
[6:15:30 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:15:32 PM] ✅ ✅ metadata citation 2 closed successfully
[6:15:32 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 15)
[6:15:32 PM] ℹ️ Extracting citation 3/3 for "Finding Christmas"
[6:15:32 PM] ℹ️ Processing metadata citation 3 for "Finding Christmas"
[6:15:33 PM] ℹ️ Clicking to OPEN metadata citation 3
[6:15:35 PM] ℹ️ Extracted citation text (first 100 chars): comes a heartwarming new Christmas story...
[6:15:35 PM] ℹ️ Clicking to CLOSE metadata citation 3
[6:15:37 PM] ✅ ✅ metadata citation 3 closed successfully
[6:15:37 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 40)
[6:15:37 PM] ℹ️ Collapsing book section: "Finding Christmas"
[6:15:38 PM] ✅ Book section collapsed: "Finding Christmas"
[6:15:38 PM] ✅ ✅ Completed processing "Finding Christmas" - 3 citations extracted
[6:15:38 PM] 📋 📖 PROCESSING BOOK 2: "How the Dukes Stole Christmas"
[6:15:38 PM] ℹ️ Expanding book section: "How the Dukes Stole Christmas"
[6:15:41 PM] ✅ Book section expanded: "How the Dukes Stole Christmas"
[6:15:41 PM] ℹ️ Expanding "Why this book is the match" section for: "How the Dukes Stole Christmas"
[6:15:44 PM] ✅ "Why this book is the match" section expanded for: "How the Dukes Stole Christmas"
[6:15:44 PM] ℹ️ Found 3 citation buttons for "How the Dukes Stole Christmas"
[6:15:44 PM] ℹ️ Extracting citation 1/3 for "How the Dukes Stole Christmas"
[6:15:44 PM] ℹ️ Processing metadata citation 1 for "How the Dukes Stole Christmas"
[6:15:45 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:15:48 PM] ℹ️ Extracted citation text (first 100 chars): Book Title: How the Dukes Stole Christmas: A Christmas Romance Anthology...
[6:15:48 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:15:49 PM] ✅ ✅ metadata citation 1 closed successfully
[6:15:49 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 72)
[6:15:49 PM] ℹ️ Extracting citation 2/3 for "How the Dukes Stole Christmas"
[6:15:49 PM] ℹ️ Processing metadata citation 2 for "How the Dukes Stole Christmas"
[6:15:50 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:15:52 PM] ℹ️ Extracted citation text (first 100 chars): christmas books...
[6:15:52 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:15:54 PM] ✅ ✅ metadata citation 2 closed successfully
[6:15:54 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 15)
[6:15:54 PM] ℹ️ Extracting citation 3/3 for "How the Dukes Stole Christmas"
[6:15:54 PM] ℹ️ Processing metadata citation 3 for "How the Dukes Stole Christmas"
[6:15:55 PM] ℹ️ Clicking to OPEN metadata citation 3
[6:15:57 PM] ℹ️ Extracted citation text (first 100 chars): Christmas Romance Anthology...
[6:15:57 PM] ℹ️ Clicking to CLOSE metadata citation 3
[6:15:58 PM] ✅ ✅ metadata citation 3 closed successfully
[6:15:58 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 27)
[6:15:58 PM] ℹ️ Collapsing book section: "How the Dukes Stole Christmas"
[6:16:00 PM] ✅ Book section collapsed: "How the Dukes Stole Christmas"
[6:16:00 PM] ✅ ✅ Completed processing "How the Dukes Stole Christmas" - 3 citations extracted
[6:16:00 PM] 📋 📖 PROCESSING BOOK 3: "The First Christmas"
[6:16:00 PM] ℹ️ Expanding book section: "The First Christmas"
[6:16:03 PM] ✅ Book section expanded: "The First Christmas"
[6:16:03 PM] ℹ️ Expanding "Why this book is the match" section for: "The First Christmas"
[6:16:06 PM] ✅ "Why this book is the match" section expanded for: "The First Christmas"
[6:16:06 PM] ℹ️ Found 3 citation buttons for "The First Christmas"
[6:16:06 PM] ℹ️ Extracting citation 1/3 for "The First Christmas"
[6:16:06 PM] ℹ️ Processing metadata citation 1 for "The First Christmas"
[6:16:07 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:16:09 PM] ℹ️ Extracted citation text (first 100 chars): Book Title: The First Christmas...
[6:16:09 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:16:11 PM] ✅ ✅ metadata citation 1 closed successfully
[6:16:11 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 31)
[6:16:11 PM] ℹ️ Extracting citation 2/3 for "The First Christmas"
[6:16:11 PM] ℹ️ Processing metadata citation 2 for "The First Christmas"
[6:16:12 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:16:14 PM] ℹ️ Extracted citation text (first 100 chars): christmas books...
[6:16:14 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:16:15 PM] ✅ ✅ metadata citation 2 closed successfully
[6:16:15 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 15)
[6:16:15 PM] ℹ️ Extracting citation 3/3 for "The First Christmas"
[6:16:15 PM] ℹ️ Processing manuscript citation 3 for "The First Christmas"
[6:16:16 PM] ℹ️ Clicking to OPEN manuscript citation 3
[6:16:18 PM] ℹ️ Extracted citation text (first 100 chars): The idea to write this book about the stories of the first Christmas...
[6:16:18 PM] ℹ️ Clicking to CLOSE manuscript citation 3
[6:16:20 PM] ✅ ✅ manuscript citation 3 closed successfully
[6:16:20 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 68)
[6:16:20 PM] ℹ️ Collapsing book section: "The First Christmas"
[6:16:21 PM] ✅ Book section collapsed: "The First Christmas"
[6:16:21 PM] ✅ ✅ Completed processing "The First Christmas" - 3 citations extracted
[6:16:21 PM] 📋 📖 PROCESSING BOOK 4: "Christmas 101"
[6:16:21 PM] ℹ️ Expanding book section: "Christmas 101"
[6:16:25 PM] ✅ Book section expanded: "Christmas 101"
[6:16:25 PM] ℹ️ Expanding "Why this book is the match" section for: "Christmas 101"
[6:16:28 PM] ✅ "Why this book is the match" section expanded for: "Christmas 101"
[6:16:28 PM] ℹ️ Found 3 citation buttons for "Christmas 101"
[6:16:28 PM] ℹ️ Extracting citation 1/3 for "Christmas 101"
[6:16:28 PM] ℹ️ Processing metadata citation 1 for "Christmas 101"
[6:16:29 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:16:31 PM] ℹ️ Extracted citation text (first 100 chars): Book Title: Christmas 101...
[6:16:31 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:16:32 PM] ✅ ✅ metadata citation 1 closed successfully
[6:16:32 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 25)
[6:16:32 PM] ℹ️ Extracting citation 2/3 for "Christmas 101"
[6:16:32 PM] ℹ️ Processing metadata citation 2 for "Christmas 101"
[6:16:33 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:16:35 PM] ℹ️ Extracted citation text (first 100 chars): christmas books...
[6:16:35 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:16:37 PM] ✅ ✅ metadata citation 2 closed successfully
[6:16:37 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 15)
[6:16:37 PM] ℹ️ Extracting citation 3/3 for "Christmas 101"
[6:16:37 PM] ℹ️ Processing metadata citation 3 for "Christmas 101"
[6:16:38 PM] ℹ️ Clicking to OPEN metadata citation 3
[6:16:40 PM] ℹ️ Extracted citation text (first 100 chars): holiday planning simple, fun and festive...
[6:16:40 PM] ℹ️ Clicking to CLOSE metadata citation 3
[6:16:41 PM] ✅ ✅ metadata citation 3 closed successfully
[6:16:41 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 40)
[6:16:41 PM] ℹ️ Collapsing book section: "Christmas 101"
[6:16:43 PM] ✅ Book section collapsed: "Christmas 101"
[6:16:43 PM] ✅ ✅ Completed processing "Christmas 101" - 3 citations extracted
[6:16:43 PM] 📋 📖 PROCESSING BOOK 5: "So This Is Christmas"
[6:16:43 PM] ℹ️ Expanding book section: "So This Is Christmas"
[6:16:46 PM] ✅ Book section expanded: "So This Is Christmas"
[6:16:46 PM] ℹ️ Expanding "Why this book is the match" section for: "So This Is Christmas"
[6:16:49 PM] ✅ "Why this book is the match" section expanded for: "So This Is Christmas"
[6:16:49 PM] ℹ️ Found 3 citation buttons for "So This Is Christmas"
[6:16:49 PM] ℹ️ Extracting citation 1/3 for "So This Is Christmas"
[6:16:49 PM] ℹ️ Processing metadata citation 1 for "So This Is Christmas"
[6:16:50 PM] ℹ️ Clicking to OPEN metadata citation 1
[6:16:52 PM] ℹ️ Extracted citation text (first 100 chars): Book Title: So This Is Christmas...
[6:16:52 PM] ℹ️ Clicking to CLOSE metadata citation 1
[6:16:54 PM] ✅ ✅ metadata citation 1 closed successfully
[6:16:54 PM] ℹ️ ✅ Citation 1 extracted successfully (length: 32)
[6:16:54 PM] ℹ️ Extracting citation 2/3 for "So This Is Christmas"
[6:16:54 PM] ℹ️ Processing metadata citation 2 for "So This Is Christmas"
[6:16:55 PM] ℹ️ Clicking to OPEN metadata citation 2
[6:16:57 PM] ℹ️ Extracted citation text (first 100 chars): christmas books...
[6:16:57 PM] ℹ️ Clicking to CLOSE metadata citation 2
[6:16:58 PM] ✅ ✅ metadata citation 2 closed successfully
[6:16:58 PM] ℹ️ ✅ Citation 2 extracted successfully (length: 15)
[6:16:58 PM] ℹ️ Extracting citation 3/3 for "So This Is Christmas"
[6:16:58 PM] ℹ️ Processing metadata citation 3 for "So This Is Christmas"
[6:16:59 PM] ℹ️ Clicking to OPEN metadata citation 3
[6:17:01 PM] ℹ️ Extracted citation text (first 100 chars): the country’s magical Christmas season...
[6:17:01 PM] ℹ️ Clicking to CLOSE metadata citation 3
[6:17:03 PM] ✅ ✅ metadata citation 3 closed successfully
[6:17:03 PM] ℹ️ ✅ Citation 3 extracted successfully (length: 38)
[6:17:03 PM] ℹ️ Collapsing book section: "So This Is Christmas"
[6:17:05 PM] ✅ Book section collapsed: "So This Is Christmas"
[6:17:05 PM] ✅ ✅ Completed processing "So This Is Christmas" - 3 citations extracted
[6:17:05 PM] ✅ 🎉 Successfully extracted citations from 5 books
[6:17:05 PM] ℹ️ Step 3: Validating reason-citation matches
[6:17:05 PM] 📋 🔍 VALIDATING REASON-CITATION MATCHES (80% THRESHOLD)
[6:17:05 PM] ℹ️ 📖 Validating 3 reasons for: "Finding Christmas"
[6:17:05 PM] ℹ️ Reason 1: Checking match...
[6:17:15 PM] ✅ ✅ "Finding Christmas" - Reason 1: 95% match - PASS
[6:17:15 PM] ℹ️ Reason 2: Checking match...
[6:17:15 PM] ✅ ✅ "Finding Christmas" - Reason 2: 100% match - PASS
[6:17:15 PM] ℹ️ Reason 3: Checking match...
[6:17:26 PM] ✅ ✅ "Finding Christmas" - Reason 3: 95% match - PASS
[6:17:26 PM] ℹ️ 📖 Validating 3 reasons for: "How the Dukes Stole Christmas"
[6:17:26 PM] ℹ️ Reason 1: Checking match...
[6:17:36 PM] ✅ ✅ "How the Dukes Stole Christmas" - Reason 1: 95% match - PASS
[6:17:36 PM] ℹ️ Reason 2: Checking match...
[6:17:36 PM] ✅ ✅ "How the Dukes Stole Christmas" - Reason 2: 100% match - PASS
[6:17:36 PM] ℹ️ Reason 3: Checking match...
[6:17:36 PM] ✅ ✅ "How the Dukes Stole Christmas" - Reason 3: 100% match - PASS
[6:17:36 PM] ℹ️ 📖 Validating 3 reasons for: "The First Christmas"
[6:17:36 PM] ℹ️ Reason 1: Checking match...
[6:17:36 PM] ❌ ❌ "The First Christmas" - Reason 1: 0% match - FAIL
[6:17:36 PM] ❌ ⚠️ Citation text not found in reason: "Book Title: The First Christmas"
[6:17:36 PM] ℹ️ Reason 2: Checking match...
[6:17:36 PM] ✅ ✅ "The First Christmas" - Reason 2: 100% match - PASS
[6:17:36 PM] ℹ️ Reason 3: Checking match...
[6:17:36 PM] ❌ ❌ "The First Christmas" - Reason 3: 0% match - FAIL
[6:17:36 PM] ❌ ⚠️ Citation text not found in reason: "The idea to write this book about the stories of the first Christmas"
[6:17:36 PM] ℹ️ 📖 Validating 3 reasons for: "Christmas 101"
[6:17:36 PM] ℹ️ Reason 1: Checking match...
[6:17:36 PM] ❌ ❌ "Christmas 101" - Reason 1: 0% match - FAIL
[6:17:36 PM] ❌ ⚠️ Citation text not found in reason: "Book Title: Christmas 101"
[6:17:36 PM] ℹ️ Reason 2: Checking match...
[6:17:36 PM] ✅ ✅ "Christmas 101" - Reason 2: 100% match - PASS
[6:17:36 PM] ℹ️ Reason 3: Checking match...
[6:17:36 PM] ✅ ✅ "Christmas 101" - Reason 3: 100% match - PASS
[6:17:36 PM] ℹ️ 📖 Validating 3 reasons for: "So This Is Christmas"
[6:17:36 PM] ℹ️ Reason 1: Checking match...
[6:17:36 PM] ❌ ❌ "So This Is Christmas" - Reason 1: 0% match - FAIL
[6:17:36 PM] ❌ ⚠️ Citation text not found in reason: "Book Title: So This Is Christmas"
[6:17:36 PM] ℹ️ Reason 2: Checking match...
[6:17:36 PM] ✅ ✅ "So This Is Christmas" - Reason 2: 100% match - PASS
[6:17:36 PM] ℹ️ Reason 3: Checking match...
[6:17:36 PM] ✅ ✅ "So This Is Christmas" - Reason 3: 100% match - PASS
[6:17:36 PM] 📊 📊 CITATION VALIDATION SUMMARY: 11/15 passed (73.3%)
[6:17:36 PM] ❌ 💥 REASON-CITATION VALIDATION: OVERALL FAILED
[6:17:36 PM] ℹ️ Step 4: Generating detailed report
[6:17:36 PM] 📋 📋 GENERATING CITATION VALIDATION REPORT
[6:17:36 PM] ℹ️ Report generated: 11/15 reasons passed
[6:17:36 PM] ✅ ✅ Citation validation workflow completed successfully
[6:17:36 PM] 📊 📈 OVERALL CITATION VALIDATION: 11/15 (73.3%)
[6:17:36 PM] ❌ ❌ REASON-CITATION VALIDATION: OVERALL FAILED - Only 73.3% of reasons match citations (required: 80%)
[6:17:36 PM] 📋 📋 GENERATING DETAILED CITATION VALIDATION REPORT
[6:17:36 PM] ℹ️ Generating HTML citation validation report
[6:17:36 PM] ✅ Generated HTML report for 15 reasons
[6:17:36 PM] ✅ ✅ Citation validation HTML report generated and attached
[6:17:36 PM] ✅ ✅ Citation validation plain text report attached
[6:17:36 PM] ℹ️ Checking database connectivity...
[6:17:37 PM] ✅ ✅ Database connected successfully
[6:17:37 PM] ℹ️ Database path: /home/runner/work/HC_Epub_Automation/HC_Epub_Automation/tests/test_data/database.xlsx
[6:17:37 PM] ℹ️ Total books: 10032
[6:17:37 PM] 📊 Database contains 10032 books
[6:17:37 PM] ✅ ✅ Database book count requirement met: 10032 >= 10032
[6:17:37 PM] 📋 🔍 VALIDATING EXTRACTED BOOKS INDIVIDUALLY AGAINST DATABASE
[6:17:37 PM] ℹ️ Checking book 1/5: "Finding Christmas"
[6:17:37 PM] ℹ️ Checking if book exists in database: "Finding Christmas"
[6:17:37 PM] ✅ ✅ Database match: "Finding Christmas" → "Finding Christmas"
[6:17:37 PM] ℹ️ Checking book 2/5: "How the Dukes Stole Christmas"
[6:17:37 PM] ℹ️ Checking if book exists in database: "How the Dukes Stole Christmas"
[6:17:37 PM] ✅ ✅ Database match: "How the Dukes Stole Christmas" → "How the Dukes Stole Christmas"
[6:17:37 PM] ℹ️ Checking book 3/5: "The First Christmas"
[6:17:37 PM] ℹ️ Checking if book exists in database: "The First Christmas"
[6:17:38 PM] ✅ ✅ Database match: "The First Christmas" → "The First Christmas"
[6:17:38 PM] ℹ️ Checking book 4/5: "Christmas 101"
[6:17:38 PM] ℹ️ Checking if book exists in database: "Christmas 101"
[6:17:38 PM] ✅ ✅ Database match: "Christmas 101" → "Christmas 101"
[6:17:38 PM] ℹ️ Checking book 5/5: "So This Is Christmas"
[6:17:38 PM] ℹ️ Checking if book exists in database: "So This Is Christmas"
[6:17:38 PM] ✅ ✅ Database match: "So This Is Christmas" → "So This Is Christmas"
[6:17:38 PM] 📊 INDIVIDUAL DATABASE VALIDATION SUMMARY: 5/5 books found in database
[6:17:38 PM] ✅ 🎉 ALL BOOKS VALIDATED SUCCESSFULLY AGAINST DATABASE!
[6:17:38 PM] 📋 🤖 VALIDATING RESPONSE RELEVANCE WITH GEMINI AI - PER BOOK ANALYSIS
[6:17:38 PM] ℹ️ Starting Gemini AI per-book relevance validation...
[6:17:38 PM] 📋 🤖 ANALYZING RESPONSE RELEVANCE WITH GEMINI AI - PER BOOK ANALYSIS
[6:17:38 PM] ℹ️ Starting Gemini AI analysis with retry logic...
[6:17:44 PM] ⚠ ⚠️ GEMINI VALIDATION: Using fallback analysis due to API issues
[6:17:44 PM] ℹ️ Overall Score: 70% (Fallback)
[6:17:44 PM] 📋 📖 ANALYSIS FOR: "Finding Christmas" - Score: 70%
[6:17:44 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ 💡 Book-specific improvements:
[6:17:44 PM] ℹ️ - Retry analysis when service is available
[6:17:44 PM] 📋 📖 ANALYSIS FOR: "How the Dukes Stole Christmas" - Score: 70%
[6:17:44 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ 💡 Book-specific improvements:
[6:17:44 PM] ℹ️ - Retry analysis when service is available
[6:17:44 PM] 📋 📖 ANALYSIS FOR: "The First Christmas" - Score: 70%
[6:17:44 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ 💡 Book-specific improvements:
[6:17:44 PM] ℹ️ - Retry analysis when service is available
[6:17:44 PM] 📋 📖 ANALYSIS FOR: "Christmas 101" - Score: 70%
[6:17:44 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ 💡 Book-specific improvements:
[6:17:44 PM] ℹ️ - Retry analysis when service is available
[6:17:44 PM] 📋 📖 ANALYSIS FOR: "So This Is Christmas" - Score: 70%
[6:17:44 PM] ℹ️ ❌ Author Information: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Publishing Date: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Why Match Explanations: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ ❌ Relevance Scores: 70% - Analysis unavailable - API error
[6:17:44 PM] ℹ️ 💡 Book-specific improvements:
[6:17:44 PM] ℹ️ - Retry analysis when service is available
[6:17:44 PM] ℹ️ 📊 SUMMARY FEEDBACK:
[6:17:44 PM] ℹ️ • AI analysis service is temporarily overloaded
[6:17:44 PM] ℹ️ • Using fallback analysis with neutral scores
[6:17:44 PM] ℹ️ • Error: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-pro:generateContent: [429 Too Many Requests] You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits.
* Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests, limit: 2
Please retry in 15.479670081s. [{"@type":"type.googleapis.com/google.rpc.QuotaFailure","violations":[{"quotaMetric":"generativelanguage.googleapis.com/generate_content_free_tier_requests","quotaId":"GenerateRequestsPerMinutePerProjectPerModel-FreeTier","quotaDimensions":{"location":"global","model":"gemini-2.5-pro"},"quotaValue":"2"}]},{"@type":"type.googleapis.com/google.rpc.Help","links":[{"description":"Learn more about Gemini API quotas","url":"https://ai.google.dev/gemini-api/docs/rate-limits"}]},{"@type":"type.googleapis.com/google.rpc.RetryInfo","retryDelay":"15s"}]
[6:17:44 PM] ❌ ❌ GEMINI AI VALIDATION FAILED: 70% relevance score (required: 80%)
[6:17:44 PM] ℹ️ Continuing execution due to soft assertion...